Alain Kelder is a Giant Dork

Script to individually terminate “stuck” web server threads without restarting Apache

At work, a misbehaving web app was causing Apache worker threads to get indefinitely "stuck" in "W" state (Sending Reply), not returning anything to client, but taking up a chunk of CPU (often maxing a single thread at 99%). As additional requests for problematic pages came in, they would eventually tie up all available processor capacity, leading to high load averages (worst it got was a load average of 90 while googlebot was crawling) and a very sluggish server. Problem would get "resolved" by restarting Apache, but of course that killed all the clients, so needed a more targeted solution.

While the developer is investigating, as a stop gap measure, came up with this script to identify and individually terminate these stuck Apache workers and log the request URL for later analysis. At first, I had monit run it whenever Apache CPU usage exceeded a threshold, but that wasn't often enough, so now it's called by crontab at 1 min intervals. If no Apache worker threads match the pattern, the script silently exits. So far, it's worked really well with zero false positives, though of course this is a temporary stop gap measure and not something I'd run normally.

Example notification sent and log entry made by script when Apache workers are stopped:

--------------------------------
 Stopped on 2012-08-28 22:38:01
--------------------------------
Srv   PID    Acc         M  CPU    SS  Req Conn Child  Slot Client        VHost             Request                   
13-4  22979  4/87/618    W  66.29  81   0  0.0  0.06  1.79  66.249.68.16  host.example.com  GET  /path/to/page  HTTP/1.1
0-4   25369  75/86/2054  W  42.95  79   0  0.0  0.00  9.28  66.249.68.16  host.example.com  GET  /path/to/page  HTTP/1.1
15-3  2133   86/346/449  W  67.24  72   0  0.0  0.49  1.15  66.249.68.16  host.example.com  GET  /path/to/page  HTTP/1.1     

And here's the script:

#/bin/bash

GetAllWorkers()
{
 AllWorkers=$(apache2ctl fullstatus | awk '/^Srv /,/^$/ {print}')
}

GetStuckWorkers()
{
 StuckWorkers=$(echo "$AllWorkers" | awk '$4 == "W" && $6 > 60 && $7 == 0 && $8 == "0.0" {print}')
 header=$(echo "$AllWorkers" | head -n 1)
}

GetStuckPIDs()
{
 StuckPIDs=$(echo "$AllWorkers" | awk '$4 == "W" && $6 > 60 && $7 == 0 && $8 == "0.0" {print$2}')
}

Show()
{
 echo "--------------------------------"
 echo " Stopped on $(date +%F\ %T)"
 echo "--------------------------------"
 echo "$header"
 echo "$StuckWorkers"
}

GetAllWorkers && GetStuckPIDs
if [ -n "$StuckPIDs" ]; then
  for PID in $StuckPIDs; do
    echo stopping $PID with SIGTERM
    kill $PID
  done
  GetStuckWorkers
  Show | mail -s "$(basename $0) executed on $(hostname -s)" root
  Show >> /path/to/ksaw.log
fi